Return of the Nuggets – November 2023

Charles F. Vardeman II

Center for Research Computing, University of Notre Dame

2023-11-17

How to Prompt?

Prompting Guide

Prompt Engineering Guide

OpenAI Cookbook

OpenAI Cookbook

LLMOps Engineering

A Survey of Techniques for Maximizing LLM Performance

LLMOps Engineering

A Survey of Techniques for Maximizing LLM Performance

Framework for architecture design of LLM Based Agents

Wang, Lei, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, et al. 2023. “A Survey on Large Language Model Based Autonomous Agents.” arXiv. http://arxiv.org/abs/2308.11432.

Unit Testing of LLMs

A Survey of Techniques for Maximizing LLM Performance

Prompt Engineering is about adding context!

KGs for Context

Sequeda, Juan, Dean Allemang, and Bryon Jacob. 2023. “A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model’s Accuracy for Question Answering on Enterprise SQL Databases.” arXiv. http://arxiv.org/abs/2311.07509.

KGs for Context

Sequeda, Juan, Dean Allemang, and Bryon Jacob. 2023. “A Benchmark to Understand the Role of Knowledge Graphs on Large Language Model’s Accuracy for Question Answering on Enterprise SQL Databases.” arXiv. http://arxiv.org/abs/2311.07509.

Information Extraction for RAG (Tool Use)

Xu, Silei, Shicheng Liu, Theo Culhane, Elizaveta Pertseva, Meng-Hsi Wu, Sina J. Semnani, and Monica S. Lam. 2023. “Fine-Tuned LLMs Know More, Hallucinate Less with Few-Shot Sequence-to-Sequence Semantic Parsing over Wikidata.” arXiv. http://arxiv.org/abs/2305.14202.

Prompting Patterns for RAG – Planning and Action

Prasad, Archiki, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. 2023. “ADaPT: As-Needed Decomposition and Planning with Language Models.” arXiv. https://doi.org/10.48550/arXiv.2311.05772.

Training and Fine Tuning

Textbooks are all you need II: phi-1.5

Y. Li, S. Bubeck, R. Eldan, A. Del Giorno, S. Gunasekar, and Y. T. Lee, “Textbooks Are All You Need II: phi-1.5 technical report.” arXiv, Sep. 11, 2023. Accessed: Sep. 12, 2023. [Online]. Available: http://arxiv.org/abs/2309.05463

Maybe we need more than Textbooks?

A “Curriculum” for Logic?

Feng, Jiazhan, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, and Weizhu Chen. 2023. “Language Models Can Be Logical Solvers.” arXiv. http://arxiv.org/abs/2311.06158.

A “Curriculum” for Logic?

Feng, Jiazhan, Ruochen Xu, Junheng Hao, Hiteshi Sharma, Yelong Shen, Dongyan Zhao, and Weizhu Chen. 2023. “Language Models Can Be Logical Solvers.” arXiv. http://arxiv.org/abs/2311.06158.